Although it has exhibited explosive growth and extensively impacted the worlds of information and commerce, the globally accessible computer network known as the Internet has effectively become an unstructured victim of itself. Internet information usage has largely lost its utility because traditional search engines can neither access the vast available information pool nor qualify it to provide access to back-end systems, silos of data, and a myriad of devices. In short, the infrastructure that provides the traditional “World Wide Web” of surfing with browsers is not suitable for future information needs.
The World Wide Web itself represents only a small fraction of the total information that exists, because organizations around the world convert only a small portion of their data to Web-accessible pages. The remainder is buried in the IT infrastructure of those organizations as database tables, images, document files, and other forms of media; and the vast majority of that data remains inaccessible and unmined with current search technology.
Other factors further hamper the ability to access these data. It is primarily private, secured behind layers of encryption and firewalls; and retrieving it requires authentication, translation, and complex navigation. The data are distributed among the IT infrastructure of a great many different organizations, all of which have their own network topologies, programming interfaces, data structures, and profiles for hardware and software. In effect, the organizations' infrastructures cannot “talk” to one another because they all “speak a different language” and the network of paths to the data is complex.
Tools must be implemented to translate between those myriad IT infrastructures—to allow a common data model and create access for data exchange. Without these tools, all that data, which currently dwarf the data available on the World Wide Web, are inaccessible and unmined. The expansion of data has been and will continue to be exponential in nature, predictably having doubled in size every couple of years.
Part of the fuel of this growth is that the number of devices that access the Internet will undergo explosive growth in the next decade. A Cisco report estimates that, as of 2012, 8.7 billion devices were connected to the Internet: primarily desktop and laptop computers, tablets, and phones. Based on another Cisco report, an estimate by Morgan Stanley is that the 8.7 billion devices will balloon to 75 billion by 2020—a nearly nine-fold increase in the next seven years that amounts to 9.4 Internet-connected devices for each of the eight billion people on Earth in 2020. All of those 75 billion devices must be measured, analyzed, and acted upon with software.
These devices will be more than laptops and phones. The market will soon be dominated by sensors, actuators, and other similar devices that fundamentally alter the way people live and work. A sensor on a store shelf in a supermarket will identify the lack of a certain product, thereby starting an automated process that alerts the entire manufacturing and supply chain and starts automated processes at multiple other organizations. Smart homes will identify family members and adjust the room temperature based on their preferences as the family members move throughout the house. The possibilities are almost without limit.
The result of this explosive growth in data and devices is a worsening of the current state of distributed, inaccessible, unorganized, uncategorized, and unmined data. The cloud will evolve into a massive data mesh or knowledge fabric. Software that can organize and connect the furthest and most obscure reaches of this fabric is needed, and the need will increase with the exponential increase in data and devices over the next decade.
There is a pressing need for software to enable the access of—and applications needed to process—this “big data” volume so that the enormous number of different organizations can participate in the data exchange, in effect establishing a worldwide network topology. In this way, devices and software can talk to, interact with, and exchange data with devices and software in a vastly distributed world.